Polling

:::

• We want to estimate the proportion of all registered voters — our population — who will vote.
• To do this, we use the proportion of polled voters — our sample — who said they would.
• When the election occurs, we get to see who turns out to vote.
  • 5.05M people, or roughly 70% of registered voters, actually vote.

The Sampling Distribution of our Estimate

\[ \begin{array}{r|rr|rr|r|rr|r} \text{call} & 1 & & 2 & & \dots & 625 & & \\ \text{poll} & J_1 & Y_1 & J_2 & Y_2 & \dots & J_{625} & Y_{625} & \overline{Y}_{625} \\ \hline \color[RGB]{7,59,76}1 & \color[RGB]{7,59,76}869369 & \color[RGB]{7,59,76}1 & \color[RGB]{7,59,76}4428455 & \color[RGB]{7,59,76}1 & \color[RGB]{7,59,76}\dots & \color[RGB]{7,59,76}1268868 & \color[RGB]{7,59,76}1 & \color[RGB]{7,59,76}0.68 \\ \color[RGB]{239,71,111}2 & \color[RGB]{239,71,111}600481 & \color[RGB]{239,71,111}0 & \color[RGB]{239,71,111}6793745 & \color[RGB]{239,71,111}1 & \color[RGB]{239,71,111}\dots & \color[RGB]{239,71,111}1377933 & \color[RGB]{239,71,111}1 & \color[RGB]{239,71,111}0.71 \\ \color[RGB]{17,138,178}3 & \color[RGB]{17,138,178}3830847 & \color[RGB]{17,138,178}1 & \color[RGB]{17,138,178}5887416 & \color[RGB]{17,138,178}1 & \color[RGB]{17,138,178}\dots & \color[RGB]{17,138,178}4706637 & \color[RGB]{17,138,178}1 & \color[RGB]{17,138,178}0.70 \\ {\vdots} & {\vdots} & {\vdots} & {\vdots} & {\vdots} & {\vdots} & {\vdots} & {\vdots} & {\vdots} \\ \color[RGB]{6,214,160}1M & \color[RGB]{6,214,160}2533350 & \color[RGB]{6,214,160}1 & \color[RGB]{6,214,160}5539770 & \color[RGB]{6,214,160}1 & \color[RGB]{6,214,160}\dots & \color[RGB]{6,214,160}7068692 & \color[RGB]{6,214,160}1 & \color[RGB]{6,214,160}0.71 \\ {\vdots} & {\vdots} & {\vdots} & {\vdots} & {\vdots} & {\vdots} & {\vdots} & {\vdots} & {\vdots} \\ \end{array} \]

Interval Estimation

• To give a sense of what this tells us about the population proportion, we can report an interval estimate.
  • i.e., we can draw ‘arms’ on our point estimate \(\bar Y_{625}\).
  • Here, we’ve used arms of width .02. i.e., we’ve drawn in the interval \(\bar Y_{625} \pm 0.01\).
• But this is only useful if we have a sense of how likely this interval is to be ‘right.’
• That is, how often, when we report and interval, it contains the population proportion.
• We call this a coverage probability. In this case, it’s bad. It’s about 43%.
• If you run polls and report intervals like this, you’ll make misleading claims about turnout 57% of the time.
• That’s not a good look. To fix this, we need to calibrate our interval estimates.

A Useful Counting Trick

• We don’t really need to look at every interval—and its arms—to calculate coverage.
• Instead, we can draw arms of the same width around the population proportion.
• An interval estimate covers the population proportion if and only if …
• the corresponding point estimate is between the population proportion’s arms.

A Useful Counting Trick

• So all we need to do is count how many of our point estimates are between the population proportion’s arms.
• Or, equivalently, calculate the mass of our histogram between the population proportion’s arms.
• That is, we calculate the area of the verticals bars in the shaded region.
• You can see it’s roughly half the area of the histogram itself. Hence, roughly 50% coverage.

Calibrating Interval Estimates

• To increase the coverage of our intervals estimates, we need to make the arms wider.
• To calibrate this width to give us 95% coverage—which is the convention—we can use our counting trick.
• We expand our shaded region until it covers 95% of the histogram’s mass, i.e., 95% of our point estimates.
• If we use intervals of this width around our point estimates, we’ll get 95% coverage.

Calibrating Interval Estimates

• To increase the coverage of our intervals estimates, we need to make the arms wider.
• To calibrate this width to give us 95% coverage—which is the convention—we can use our counting trick.
• We expand our shaded region until it covers 95% of the histogram’s mass, i.e., 95% of our point estimates.
• If we use intervals of this width around our point estimates, we’ll get 95% coverage.

The Problem

• Unless we’ve seen the population, i.e. unless it’s after the election, we can’t run simulated polls.
• That means we can’t calibrate our interval using the sampling distribution of our point estimate.
• That’s something we don’t know. The good news is that it’s something we can estimate.
• So we’ll actually do calibration using an estimate of the sampling distribution.

Calibration using a Sampling Distribution Estimate

• Here we’ve estimated the sampling distribution — we’ll get to how — and used it to find an interval width.
• In particular, we’ve used a width which …
  • if we draw arms around the mean of our estimated sampling distribution
  • will cover 95% of draws from our estimated sampling distribution.
• The one apparent sticking point is that our estimated sampling distribution is a bit off-center.
• Its mean is a bit lower than the population proportion. It’s actually our sample proportion \(\bar Y\).
• It turns out that this doesn’t matter much. Why?

Calibration Comparison

• It doesn’t matter because we’re not putting arms on draws from the estimated sampling distribution.
• We’re putting arms on our point estimate, which is a draw from the actual sampling distribution.
• To do this, we’re using the width—but not the center—of the estimated sampling distribution.
• And it works because that’s very close to the width we’d get from the actual sampling distribution.
• Above, I’ve drawn in shaded regions corresponding to two versions of the the population proportion’s arms.
  • The green region is the one we get from the actual sampling distribution. We can’t use this.
  • The red region is the one we get from the estimated sampling distribution. We can — and do — use this.
• And I’ve drawn in interval estimates based on our poll using both widths. They’re almost the same.

Our Samping Distribution is Binomial

• To estimate the sampling distribution, it helps to know something about the actual sampling distribution.
  • It’s the distribution of the proportion heads in 625 flips of a coin with probability \(p\approx 0.70\) of heads.
  • where \(p \approx 0.70\) is the proportion of registered voters who will vote.
• There’s a name for that: it’s the binomial distribution.
  • To be more precise, the binomial distribution is for the number of heads.
  • And we scale that by \(1/625\) to get a distribution for the proportion.

\[ 625 \times \bar{Y}_{625} = \sum_{i=1}^{625} Y_i \sim \text{Binomial}(625, p) \qfor p = \bar y_{7.23M} = \frac{1}{7.23M}\sum_{j=1}^{7.23M} y_j \]

• There’s a formula for this probability, so we don’t have to draw samples and histogram if we don’t want to.
• In R, it’s dbinom(x, n, p), where x is the number of heads, n is the number of flips, and p is the probability of heads.

Estimating the Sampling Distribution

\[ 625 \times \bar{Y}_{625} = \sum_{i=1}^{625} Y_i \sim \text{Binomial}(625, p) \qfor p = \bar y_{7.23M} = \frac{1}{7.23M}\sum_{j=1}^{7.23M} y_j \]

• What we don’t know is \(p=\bar y\), the proportion of registered voters who turn out.
• But that’s what we’re using our sample proportion \(\bar Y\) to estimate.
• And we can just plug that in to get an estimate of the sampling distribution.
• To draw samples from this estimated sampling distribution, we use rbinom.

samples = rbinom(10000, 625, mean(Y)) / 625

• That is exactly how I got the estimated sampling distribution histogram I was using for calibration.
  • If we prefer, we can calibrate using the actual estimated binomial distribution, i.e., using dbinom.
  • We’ll explore this a bit at some point in the future, but it doesn’t make much difference.

Each of Our Friends Gets Their Own Estimate

• This means each of our friends estimates a different width for their interval.
• But they’re all close to the width we get using the actual sampling distribution. You can see it above.
  • I’ve plotted the intervals they’d use — based on their sampling distribution estimates — in bold colors.
  • And the intervals they wish they could — based on the actual sampling distribution — more faintly.

Evidence That It Works

• We see the same thing when we look at 100 polls.
• Our friends’ sampling distribution estimates are all close to the actual sampling distribution.
• The resulting interval-widths are near-indistinguishable from the width we got using the sampling distribution.
  • The estimated widths are shown as points with arms.
  • That sampling distribution’s actual width is the space between the dotted lines.
• You can see estimating the width like this works reliably.

Why Does it Work?

• Why are we getting a good estimate of the sampling distribution?
• The Binomial distribution is continuous as a function of \(p\)—when \(p\) changes little, the distribution changes little.
• This means that, if we’re not far off in the proportion, the estimated and actual sampling distributions are similar.
• The relevant difference (after centering) is even smaller because the way the binomial changes is mostly location.
• Here I’m showing three estimates of the sampling distribution based on three polls.
  • with centering (right) and without (left).
• In particular, polls giving estimates at the 2.5th, 50th, and 97.5th percentiles of the actual sampling distribution.

The Estimate Works. Why Does it Work?

• You can think of this as a sort of ‘confidence interval’ for our estimate of the sampling distribution.
  • 95% of the time, you’ll get an estimate somewhere between the red and blue ones.
  • And, as a result, a width for your interval estimate somewhere between the red and blue widths.
• As you can see, the widths we get barely change. What changes is the location.
• But because we’re using calibrated 95% intervals, we still cover the population proportion 95% of the time.
  • i.e., we cover it when our sample proportion is between the red and blue ones.
• Or, accounting for slight underestimates of the sampling distribution’s width, almost cover it.

The Bootstrap Interpretation

• There’s another way to think about this.
• The estimated sampling distribution is \(\text{Binomial}(n,p)/n\) for \(n=625\) and \(p=\bar Y=444/625 \approx 0.71\).
• That is, it’s the distribution of the proportion of heads in …
  • 625 flips of a coin
  • that has probability \(\bar Y=444/625\) of being heads.
• What’s the distribution of the proportion of yeses if we poll a population of 625 people …
• … among whom the proportion of yeses is exactly \(\bar Y = 444 / 625\)?

The Bootstrap Interpretation

• It’s the same distribution. It’s \(\text{Binomial}(n,p)/n\).
• That means we can get a draw from our estimated sampling distribution by
• running a ‘poll’ of the people in our sample, i.e.
  • rolling a 625-sided die 625 times
  • ‘calling’ the corresponding person in our sample
  • and counting up the yeses we hear
• We call the sample we get from this new ‘poll’ a bootstrap sample.

The Bootstrap Interpretation

• If we plot our voters on a map, you can see the idea in visual terms.
  • On the left, we have the population.
  • In the middle, we have our sample. It’s drawn, with replacement, from the population.
  • On the right, we have something else. A new sample drawn, with replacement, from the sample.
• Each ‘call’ that a person receives increases the size of their dot: \(\text{circle area} \propto \text{number of calls}\).
• In the sample, even though it’s drawn with replacement, all dots are the same size.
  • Because we draw from such a large population, nobody gets called twice.
• In the bootstrap sample, dots vary in size.
  • Because we draw \(n\) people from a sample of size \(n\), it’s almost impossible not to call somebody twice..

Bootstrapping Our Turnout Estimate

meanY.boot = array(dim=10000)
for(rr in 1:10000) {
    I = sample(1:n, n, replace=TRUE)
    meanY.boot[rr] = sum(Y[I])/n
}

ggplot() + geom_bar(aes(x=meanY.boot, y=after_stat(prop)), alpha=.3) + 
           geom_line(aes(x=(0:n)/n, y=dbinom(0:n, n, mean(Y))), color='black') + 
           xlim(.6,.8)